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(57) ABSTRACT 

The invention presents The Estimated Spectrum Adaptive 
Postfilter (ESAP) and the Iterative Prepost Filter (1PF) 
algorithms. These algorithms model a number of image- 
adaptive post-filtering and pre-post filtering methods. They 
are designed to minimize Discrete Cosine Transform (DCT) 
blocking distortion caused when images are highly com- 
pressed with the Joint Photographic Expert Group (JPEG) 
standard. The ESAP and the IPF techniques of the present 
invention minimize the mean square error (MSE) to improve 
the objective and subjective quality of low-bit-rate JPEG 
gray-scale images while simultaneously enhancing percep- 
tual visual quality with respect to baseline JPEG images. 

10 Claims, 8 Drawing Sheets 





U.S. Patent 


Jul. 6, 2004 


Sheet 1 of 8 


US 6,760,487 B1 



FIG. 









U.S. Patent 


Jul. 6, 2004 


Sheet 2 of 8 


US 6,760,487 B1 



3 






U.S. Patent 


Jul. 6, 2004 


FIG. 3(a) 

64 x 64 NIVF 




FIG. 3(c) 

64 x64 NIHF 


Sheet 3 of 8 


US 6,760,487 B1 







U.S. Patent 


Jul. 6, 2004 


Sheet 5 of 8 


US 6,760,487 B1 


FIG. 5(a) FIG. 5(b) 



JPEG Lena @ 0.25 8 : PP, 3168 dB AQ-ESAP Lera $ 0.25 BPP, 33.01 dB 



JPEG Bartata @ 0.5 BPP. 28,27 dB JO-ESAP Barbara § 0.5 BPP, 31 23 dB 
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FIG. 7(d) FIG. 7(e) 

Decoded DRC image x r DRE image x. 




FIG. 7(f) FIG. 7(g) 
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ESTIMATED SPECTRUM ADAPTIVE 
POSTFILTER AND THE ITERATIVE 
PREPOST FILTERING ALGIRIGHMS 

This application claims the benefit of Provisional Appli- 
cation No. 60/130,475, filed Apr. 22, 1999. 

ORIGIN OF THE INVENTION 

The invention described herein was made by employee of 
the United States Government. The invention may be manu- 
factured and used by or for the governmental purposes 
without the payment of royalties thereon or therefor. 

BACKGROUND OF THE INVENTION 

The present invention relates generally to image 
processing, and particularly to image enhancement follow- 
ing JPEG image compression and decompression. Prior 
methods for enhancing compressed images are beset with 
disadvantages or limitations. To achieve acceptable com- 
pression rates, the JPEG standard adopted a lossy compres- 
sion technique. Furthermore, the JPEG, MPEG, H.261 and 
HDITV image and video coding algorithms are severely 
impaired by blocking artifacts when operating at low bit 
rates. A compressed image becomes distorted when com- 
pressed by lossy methods. Distortion in the image can be 
confined to hidden areas of the image by carefully imple- 
menting the JPEG standard and several enhancement meth- 
ods for reducing the distortion have been attempted. 
However, in some regions of the image these prior methods 
may further degrade the image. Prior methods for reducing 
distortion introduced by the JPEG compression algorithm 
can introduce a high-frequency artifact around high contrast 
edges. Algorithms for removing artifacts exist, but can 
mistakenly remove thin lines and smooth over important 
texture. Errors also arise between adjacent 8x8 pixel image 
blocks created by opposite-signed round-off errors in the 
two blocks. One way to remove these errors is to filter the 
image using a low-pass filter. However, other image regions 
that contain important high frequency information may 
suffer a loss in fidelity. Likewise, many other post- 
processing algorithms are most effective when they are 
selectively applied only to certain regions of an image. 
Previous techniques are mainly non-linear filtering methods 
based on local pixel statistics rather than on local frequency 
content. Prior efforts include; JPEG Cross Block Smoothing 
(CBS), Projection Onto Complex Sets (POCS), and a variety 
of nonlinear filters. These offer lower peak signal to noise 
ratio (PSNR) and lower subjective quality than the improved 
algorithms proposed in the present invention, however. 
Additional background information may be found in the 
following literature references: 

IJG JPEG Software 0 1990-95, Tom G. Lane. 

Canny Edge Detector from “X-based Image Processing 
Tools and Environment (XITE)”, S. Boe, 1994. 

JPEG image-adaptive DCT coefficient quantization software 
"adaptQ( )” as described and made available by M. 
Crouse and K. Ramchandran, “Joint Thresholding and 
Quantizer Selection for Transform Image Coding: 
Entropy-Constrained Analysis and Applications to Base- 
line JPEG,” IEEE Transactions on Image Processing, vol. 
6, No. 2. pp. 205-297, February 1997. 

"Theory and Applications of the Estimated Spectrum Adap- 
tive Post-filter”, Ph.D. thesis, Georgia Institute of Tech- 
nology. © 1998 Irving Linares, Pages 1-113 May, 1998. 

SUMMARY OF THE INVENTION 
ESAP and IPF algorithms of the present invention sig- 
nificantly reduce the blocking artifacts resulting from high 
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compression by reusing the DCT coefficient local frequency 
characteristics to control a pixel-adaptive non-linear post- 
filter or a pre-post filtering system. This allows more com- 
pression and better quality compression when compared 
5 with the default JPEG compression parameters. These algo- 
rithms provide a new frequency-based pixel-adaptive filter- 
ing algorithm which may be used to enhance de-compressed 
JPEG images and to enhance MPEG sequences for Internet 
or HDTV video applications. 

10 One object of the present invention is to reduce blocking 
artifacts in low-bit-rate JPEG images, particularly in sys- 
tems wherein image pixels are coded using bit to pixel ratios 
less than or equal to 0.25 bits per pixel. 

Another object is to improve PSNR and perceptual quality 
15 of coded images simultaneously and to demonstrate the 
feasibility of extending still image pre- and post-processing 
concepts to MPEG and HDTV standards. 

A further object is to suggest use of ESAP/IPF algorithms 
to improve very low rate MPEG video sequences such as 
,, 0 those processed by popular 28.8 k-56 k modems generally 
used with web browser streaming video viewers such as 
RealPlayer™ or Streamworks, for example. 

ESAP relies on DFT analysis of the DCT and is compliant 
with the coded stream syntax of the Independent JPEG 
25 Group (IJG) Version 5b Software. At the decoder, ESAP 
estimates the 2-D pixel-adaptive bandwidths directly from 
the dequantized DCT coefficient. These coefficients are used 
to control a 2-D spatially adaptive non-linear post-filter. The 
algorithm optionally performs directional filtering parallel to 
30 the edges with no filtering across the edges. Post-filtering 
images show minimal blurring of their true edges while 
blocking is significantly removed. 

IPF is based on the concepts of dbx audio noise reduction. 
IPF is a pre-post filtering system that uses inverse pair 2D 
35 filters for high frequency pre-emphasis before encoding and 
high frequency de-emphasis after decoding. Convergence to 
a unique minimum mean square error (MMSE) is possible 
for fixed quantization matrices. However, convergence can 
not be guaranteed when image-adaptive DCT quantization is 
40 jointly optimized under pre-post filtering. 

The ESAP methods of the present invention have been 
successfully applied to JPEG color images, synthetic aper- 
ture radar (SAR) images, and image sequences. Typical 
PSNR improvement depends on the image, the encoding 
45 method, and the bit rate. For 512x512 8-BPP gray-scale 
images improvement in the range between 0.5-3.2 dB over 
baseline JPEG has been observed. Adaptive quantization has 
been observed to improve 5.6 dB for 1008x1008 8-BPS 
SAR images at 4-BPP over baseline JPEG. A comparison of 
50 all the treated techniques is presented at the conclusion of 
thesis document (Irving Linares, Theory and Applications of 
the Estimated Spectrum Adaptive Post-filter, Georgia Insti- 
tute of Technology. © 1998 Irving Linares, Pages 1-113 
May, 1998) incorporated herein by reference. ESAP and IPF 
55 algorithms of the present invention significantly reduce 
blocking artifacts resulting from high compression by reus- 
ing the DCT coefficient local frequency characteristics to 
control a pixel-adaptive non-linear post-filter or a pre-post 
filtering system. This allows more compression and/or better 
60 quality compression when compared with the default JPEG 
compression parameters alone. The subjective quality of 
image improvement is more clearly shown by means of 
foregoing software-simulated results — including several 
image samples. The ESAP and IPF algorithms may also 
65 improve the visual quality of DCT coded images, such as 
JPEG images and may be extended to MPEG video 
sequences, since MPEG is largely a DOT coder. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block representation of the ESAP algorithm — 
which includes or employs a JPEG digital image 
compression-decompression system . 

FIG. 2 is a Table comparing waveforms: DCT basis 
functions L[n], Fourier transforms |F ( fo>)|, & ideal Low-Pass 
Filter forms LPF^, respectively. 

FIG. 3(a) is a non-interpolated vertical frequency (NIVF) 
image. 

FIG. 3(b) is an interpolated vertical frequency (IVF) 
image. 

FIG. 3(c) is the non-interpolated horizontal frequency 
(NIHF) image. 

FIG. 3(d) is an interpolated vertical frequency (IVF) 
image. 

FIG. 4 describes a typical block with substantial diagonal 
frequencies for the determination of rotated bandwidths 
w e [m]. 

FIG. 5(a) shows a baseline JPEG image of Lena at 0.25 
BPP. 

FIG. 5(b) is Lena showing her corresponding adaptive 
quantization (AQ-ESAP) image at 33.01 dB PSNR. 

FIG. 5(c) is a baseline JPEG Barbara image at 0.5 BPP, 
28.27 dB PSNR. 

FIG. 5(d) is Barbara showing jointly optimized (JO- 
ESAP) version at 31.23 dB PSNR. 

FIG. 6 shows the IPF JPEG blocking noise reduction 
algorithm — which is based on dbx noise reduction. 

FIG. 1(a) shows an original 8-BPP Lena image x at IPF 
encoder. 

FIG. 7(b) is a preemphasized 8-BPP Lena image x e at IPF 
encoder. 

FIG. 7(c) is a DRC 8-BPP Lena image x r at IPF encoder. 

FIG. 1(d) is a decoded DRC 8-BPP Lena image x r at IPF 
encoder. 

FIG. 1(e) is a DRE 8-BPP Lena image x e at IPF encoder. 

FIG. 1(f) is a Deemphasized 8-BPP Lena image x at IPF 
encoder. 

FIG. 7(g) is a JPEG-IPF-ESAP 8-BPP Lena image x at 
IPF encoder. 

Tables 2(a), (b), (c), and (d) summarize the decibel (dB) 
PSNR comparative results. 

Tables 3(a) and (b) show the IPF results for the 512x512 
8-BPP Lena image. 

Tables 3(c) and (d) show the results for the 512x512 
Barbara image. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

Referring now to FIG. 1, ESAP Encoder is an acronym for 
Estimated Spectrum Adaptive Post-filter (ESAP) Encoder. 
The main components of the ESAP algorithm are shown 
including a JPEG Encoder and a JPEG Decoder. Each of 
these coders includes standard Independent JPEG Group 
(IJG) Version 5b Software. From the JPEG decoder, we 
obtain the decoded DCT coefficients used to estimate each 
pixel’s local bandwidth. This process uses 1:8 spatial inter- 
polation as will be more fully explained in sections that 
follow. The ESAP algorithm iteratively searches for the 
minimum mean square error (MMSE) of the error signal 
e(m, a,t,f,w), where m are the 2-D pixel coordinates. ITie m 
coordinates are omitted for clarity in the foregoing discus- 
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sion. Local pixel bandwidth is used to adaptively post-filter 
the decoded image. The MMSE search algorithm searches a 
4-D error surface e(cr,t,f,w) to obtain an optimal 4- tuple with 
parameters cr,t,f and w as shown in FIG. 1. 

5 The parameters are: 

a=Canny edge detector’s Gaussian standard deviation 
sigma parameter (usually in the range [0.5 . . . 2.5]). 
This controls the region of support of the edge detector. 

t=Canny edge detector’s edge strength threshold, [0 . . . 
10 255] range. 

f=Magnitude of ESAP’s DCT frequency bandwidth, 
[0.0 . . . range. In conjunction with t, this classifies 
pixels into the three categories: EDGE, NON-EDGE, 
1S or TEXTURE. 

w=directional filter region of support (wxw) in pixels, 

w=[2 . . . 16], Non-directional filters have a fixed order of 
17x17 pixels to cover four contiguous 8x8 DCT blocks. 

Each iteration requires compression, decompression and 
20 bandwidth estimation. 

Estimated Spectrum Adaptive Post-filter 

FIG. 1 shows the ESAP algorithm extension to the base- 
line JPEG coder. In the block diagram of FIG. 1, x[n] is the 
25 gray-scale input image, XL(k) are the DCT coefficients of 
the transformed image, x~[n] is the decoded JPEG image and 
x[n], is the post-filtered image. 

The meaning of error signal e(a,f,t,w) was explained 
, f) earlier [as e(a,t,f,w)] and will not be repeated here. 

To obtain better visual quality and a lower MSE, ESAP 
uses image-adaptive DCT quantization tables or Q-tables. 
This is not strictly necessary, but can improve PSNR about 
1-2 dB without increasing the bit rate. In post-processing, 
35 ESAP estimates 2-D pixel adaptive bandwidths directly 
from the dequantized DCT coefficients — without incurring 
any additional side information. Post-processing usually 
provides an additional 1 dB improvement. The ESAP algo- 
rithm combines pixel-adaptive bandwidths with directional 
40 Canny edge detectors to control a 2-D spatially-adaptive 
non-linear post-filter h m [n] to significantly reduce DCT 
blocking artifacts. Overhead required to transmit the MMSE 
post-filter parameters amounts to only two to four bytes. 
Results of our experiments outlined in the pages, tables and 
45 image samples ahead show ESAP improved PSNR up to 
3.23 dB over baseline JPEG while yielding subjective 
improvement as well. 

ESAP takes into consideration the Human Visual System 
(HVS) spatial frequency masking characteristics. Based on 
50 the HVS tolerance to quantization errors in the high- 
frequency regions, ESAP performs directional filtering par- 
allel to edges with little or no filtering across edges. Low- 
frequency non-edge regions are post-filtered with separable 
non-directional adaptive low-pass filters to minimize block- 
55 ing and restore some of the image’s natural smoothness. The 
edges are post-filtered with non-separable directional low- 
pass filters. JTie filter’s directionality helps to reduce block- 
ing along the local edge without significantly reducing 
perceived fidelity across the edge, where high-frequency 
60 quantization discontinuity errors are otherwise masked by 
the HVS perception of the edge itself. 

Fourier Transform Analysis of the DCT Basis 
Functions 

65 Each DCT basis function has a Fourier transform whose 
waveform can be explained by the modulation theorem. In 
the 1-D case, this takes the form 
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x\ri\w[n\ <=$ lj (2 jt)X ( to ) * W (co) 


Equation (7) 


Referring to FIG. 2, note that multiplication of a cosine 5 
function L{n] by an eight-point rectangular window w[n] in 
the time domain is equivalent to the circular convolution **’ 
of a sine-shaped rectangular window transform W(u>) with 
an ideal pair of impulses Jt6(m±m ) resulting in the |Fy(ot>)| 
waveforms shown. Analysis of the discrete-time Fourier 
transform of each of the DCT basis functions, in conjunction 
with FIG. 2, indicates that the ripples or side lobes of the 
spectrum of each DCT basis function, shown as dashed 
lines, are the frequency representation of the DCT blocking 
at any particular spatial frequency. The window’s width 
determines the main lobe frequency resolution and simul- 15 
taneously introduces ripples. 

The ripples represent the out-of-band DCT blocking, 
while the main lobes contain the dominant in-band signal. If 
we neglect aliasing, then reduction of the DCT blocking is 
obtained by low-pass filtering the out-of-band side-lobes 
starting at a cutoff frequency W, determined by the highest- 
frequency nonzero DCT coefficient. This analysis is readily 
extensible to the 2-D case using a separable DCT. FIG. 2 
further shows the ideal low-pass filters LPF y associated with 
each DCT basis function. Table 1 below lists their cutoff 
frequencies. 

TABLE 1 


Coefficient-block bandwidth relationship 

Highest DCT Coeff. Present 

Normalized I-D Bandwidth 0 ) c 

C 0 

0.125ji 

C> 

0.250ji 

C 2 

0.375ji 

c 3 

0.500JI 

C 4 

0.625ji 

C 5 

0.750ji 

C 6 

0.875ji 

c 7 

I.Oji 


20 


25 


Using Table 1, each block’s bandwidth is found by 
inspecting the highest 2-D nonzero coefficient. Intermediate 
zero coefficients are neglected since they do not determine 
the block’s bandwidth. For example, if the block’s highest 
coefficient is C 25 , then the vertical bandwidth is 0.375 it and 
the horizontal bandwidth is 0.750 it. The 2-D local band- 
width is centered in the middle of the block for interpolation 
purposes. This analysis generates two images each having a 
64x64 pixel bandwidth from a 512x512 pixel image. These 
are the non-interpolated vertical frequency (NIVF) image 
[FIG. 3(a)] and the non-interpolated horizontal frequency 
(NIHF) image [FIG. 3(c)]. In each of these, the gray level is 
proportional to the local horizontal or vertical bandwidth. 
Each NIF is subsequently 1:8 interpolated to obtain two 
512x512 interpolated frequency (IF) images. FIGS. 3(b) and 
(d) show the interpolated vertical frequency (IVF) and the 
interpolated horizontal frequency (IHF), respectively. To 
properly filter the image boundaries, we symmetrically 
extend or replicate the IF images m(m) and the decoded 
image x~[n]. This extends the decoded image by one 8x8 
block on each side. For example, a 512x512 pixel image 
increases to 528x528 pixels and the NIF images increase 
from 64x64 to 66x66 pixels. After the adaptive convolution 
is performed, the symmetrically extended blocks and NIF 
images serve no additional purpose and are cleared. 

Non-directional Filtering 

ESAP smoothes the current pixel of the decoded image 
x , '[m 1 ,m 2 ] with a 2-D adaptive cutoff low-pass FIR Ham- 


50 


55 


40 


ming filter h ml „ i2 [n 1 ,n 2 ] which may be directional or non- 
directional. The filter’s directionality is determined from the 
output of a Canny edge detector applied to the decoded 
image. Horizontal and vertical bandwidths are obtained 
from the IHF and IVF images, respectively. EDGE and 
NON-EDGE pixels are found by the following rule: 

/* Classify EDGE & NON-EDGE pixels 7 

for (nl=0; nl<N; nl++) 
for (n2=0; n2<N; n2++) 

if (canny_mag[nl][n2]>T && 
sqrt(Bwl[nl][n2]*Bwl[nl][n21 + Bw2[nl] 
[n21*Bw2[nl][n2])>F) 
edge[nl][n21=TRUE; 
else 

edge[nl][n2=FALSE; 

wherein: 

T is a Canny edge magnitude threshold, F is a normalized 
2-D frequency magnitude threshold, and Bw are IFs. In 
other words, if a pixel is an EDGE pixel in both the spatial 
domain and the frequency domain, then the pixel is 
declared a true image EDGE pixel and it is directionally 
post-filtered. If a pixel’s Canny magnitude is <T but its 
2-D bandwidth magnitude is >F, then it is declared a 
TEXTURE pixel and filtered with an impulse 8[nl,n2]. 
Otherwise, the pixel is declared a NON-EDGE pixel and 
is subsequently post-filtered non-directionally. 

In FIG. 1, we use the equivalent notation t,f, and ro[m] for 
T,F, and Bw respectively. 

Parameters ct and w were explained earlier. 

Directional Filtering 

Once the angular orientation 0 of an edge passing through 
a pixel x"[m] are determined using the Canny edge detector, 
we compute a rotated bandwidth tu e [m] from the original 
rectangular bandwidth estimation co[m]. FIG. 4, describes a 
typical block with a substantial quantity of diagonal fre- 
quencies. The magnitude of the highest 2-D frequency in the 
frequency plane corresponds to the perpendicular cutoff 
frequency across the edge and is given by 


p„p»min(1.0, sqrt(o) cl 2 +(O c2 2 )>r 


Equation (2) 


45 To exploit the HVS high-frequency masking 


characteristics, we fix co c 


=Jt. The cutoff frequency par- 


allel to the edge is approximately the lower of the vertical 
07 


50 


®cpar“min(o) 

cl? 


Equation (3) 


55 


Now, a non-causal 2-D rotated Hamming filter can be 
expressed as 

tlfll par, parp, «1 " 2 ]= 


h tll , pa \ sqrt(Wj 2 +rc 3 )sm('0+!.an 1 (nl/n2) ) j 1 
h ln , / , n 4sqrt(« 1 2 +« 2 )cos(0+tan _1 («l/»2 ) ) \ 


Equation (4) 


60 where 0 is measured counterclockwise with respect to the 
horizontal axis n 2 (or (a> 2 ) and the 1-D Hamming window 
LPF is given by: 


65 


h^c [«]=[sin(o) c n)/(jm)][0. 54-0.46 cos(2jt n!M)\ 

0=n=M. Equation (5) 

For NON-EDGE pixels, the above expression simplifies 
to separable filter 
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M(%, “c 2 > «i J « 2 ]=* u >d[»i]* u ,c 2 [« 2 ]- Equation (6) 

Non-directional 1-D filters of Equation (6) are pre- 
computed at program initialization and accessed as a lookup 
table during execution. ESAP’s computational complexity is 
approximately 0((MN) 2 ) multiplications and additions for 
the estimated frequency interpolation and 0((MN) 2 ) 
additions, 0((NM 2 /4) multiplications for the adaptive con- 
volution of each image (wherein NxN is the image size and 
MxM is the filter size). Equations (2) and (3) and the model 
of FIG. 4 , are based on actual separable bandwidth mea- 
surements obtained from the quantized DCT coefficients of 
rotated images using MATLAB. Note that for both direc- 
tional and non-directional filtering, the pixel- adaptive nature 
of the IF images forces us to use spatially-adaptive convo- 
lution or equivalently, a linear combiner. In other words, the 
actual implementation cannot use frequency-domain filter- 
ing. Nevertheless, the analysis presented in Section entitled 
“Fourier Transform Analysis of the DCT Basis Functions” is 
useful in determining the adaptive filter’s 2-D bandwidth, 
although the filtering operation is actually performed in the 
spatial domain. 

Referring to FIG. 6, original image x at the encoder is 
pre-emphasized with filter P(co,G), where to is the FIPF 
cutoff frequency corresponding to separable DCT bandwidth 
f of FIG. 1. Note that (<»“/) and G is the pre-emphasis gain 
in dB. The dynamic range of the resulting pre-emphasized 
signal x e is compressed into an 8-BPP range of 0-255 using 
the dynamic range compression (DRC) function. 

Then, the range-compressed signal X r of FIG. 7(c) is used 
to compute MMSE optimally quantized Q-table using the 
Lagrange multiplier minimization function adaptQw( ). 

The image is then JPEG compressed using the standard 
IJG cjpeg( ) function. The 2-D pre-post filters P(to,G) and 
D(co,G) are made of separable 1-D filters, where the dB gain 
G applies to both dimensions, P(co,G) is the pre-emphasis 
filter and D(co,G) is the de-emphasis filter. P((o,G) is 
designed to obey the pre-emphasis characteristic 

p[n] = S[n] + gh[n] 4 P(ia) = 1 + gH(u). Equation (7) 


From Equation (7) above, the dB gain is given by 

G dB = 10 log 10 (l+g) Equation (8) 

since p[n] is a non-causal even-symmetric real sequence. 
The 2-D separable pre-post filter inverse pair is given by 

P((D 1 ,(D 2 ,G : )Z)( (d 1 ,(d 2 ,G)= 

Pi(o)i,G-)P2(^2^) j Di((Oi j G)D 2 ((02 J G ; )=1. Equation (9) 

Finally, companding obeys the following equations. For 
dynamic range compression 

x r =C(x s -x mi „), Equation (10) 

and for dynamic range expansion 

x* e = (// C)x* r +x em i n . Equation (11) 

where 


C=255/(x tm , n -x anax ). Equation (12) 

At the decoder, JPEG file X,(k) is de-compressed into the 
x~ r image. Then, it is dynamic range expanded (DRE) into 
the file x' e image. Next, the image is de-emphasized with the 
filter D(m,G) to create the x" image. Finally, the image is 
ESAP post-filtered into the ximage to obtain further block 


smoothing. The ESAP filter can be inserted in any of three 
possible positions: posl, pos2, or pos3. Depending on which 
position is selected, we could create the intermediate process 
images x„ x e or x. 

5 

Along with the JPEG coded image X,-(k), four overhead 
bytes are passed to the decoder: two dynamic range com- 
pression parameters (x emin , x emax ) and two pre-post filter 
parameters (to, G). This overhead only amounts to about 
10 0.0001 BPP for a 512x512 8-BPP gray-scale image at a 32:1 
compression ratio (0.25 BPP). Please note that the encoder 
contains an internal decoder. 

15 Results for ESAP Enhancement of Baseline JPEG, 

AQ and JO Images 

To compare the objective performance of several versions 
20 of the JPEG-ESAP algorithm, we use two PSNR references: 
JPEG and the embedded zerotree wavelet (EZW). We also 
show subjective improvement, including edge preservation 
with blocking reduction for the Lena and Barbara images. 

25 FIG. 5(a) shows a baseline JPEG image of Lena, at 0.25 
BPP, 31.68 dB PSNR and FIG. 5(b) shows its corresponding 
adaptive quantization (AQ-ESAP) image at 33.01 dB PSNR. 

Similarly, FIG. 5(c) is a baseline JPEG Barbara image at 
' 0.5 BPP, 28.27. dB PSNR and FIG. 5(d) is its jointly 
optimized (JO-ESAJ 5 ) version at 31.23 dB PSNR. 

Tables 2(a), (b), (c), and (d) summarize the decibel (dB) 
55 PSNR comparative results. 

Consider, for example, the 512x512 Ixna image at 0.25 
BPP. Referring to the first two lines of Tables 2(a) and (b), 
observe that the JPEG’s cross-block smoothing (CBS) 
40 reduces the PSNR by 0.04 dB. After applying ESAP to the 
default quantized JPEG image we obtain a 1.08 dB improve- 
ment. When we pre-process the image to obtain an image- 
adaptive Q-table and post-process it with ESAP we observe 
post-filtering improvement as shown under A A q. E sap- F° r 
45 this case it is 1.33 dB. 


TABLE 2(a) 





PSNRs for 512 x 

512 Lena 

image. 



50 

BPP 

JPEG 

CBS 

ESAP 

AQ 

AQ- 

ESAP 

JO 

JO-ESAP 

EZW 


0.25 

31.68 

31.64 

32.76 

31.88 

33.01 

32.34 

33.02 

33.17 


0.50 

34.90 

34.87 

35.59 

35.48 

36.23 

35.96 

36.34 

36.28 

55 

1.00 

37.96 

37.95 

38.20 

38.88 

39.23 

39.58 

39.61 

39.55 



TABLE 2(b) 


60 

PSNR improvement over baseline JPEG for 
the 512 x 512 Lena image. 



BPP A cbs A jpeg _ esap A aq 

Aaq-esap Aj 0 A jo _ esap 

Aezw 


0.25 -0.04 1.08 0.20 

1.33 0.66 1.34 

1.49 


0.50 -0.03 0.69 0.58 

1.33 1.06 1.44 

1.38 

65 

1.00 -0.01 0.24 0.92 

1.27 1.62 1.65 

1.59 
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TABLE 2(c) TABLE 3(c)-continued 


PSNRs for 512 x 512 Barbara image. 


BPP 

JPEG 

CBS 

ESAP 

AQ 

ESAP 

JO 

JO-ESAP 

EZW 

0.25 

25.02 

25.01 

25.73 

26.02 

26.96 

26.66 

27.05 

26.77 

0.50 

28.27 

28.25 

29.44 

29.99 

31.03 

30.63 

31.23 

30.53 10 

1.00 

33.10 

33.09 

34.01 

35.22 

35.84 

35.94 

36.33 

35.14 


JPEG, IPF, EZW and SPIHT comparison 
for 512 x 512 Barbara image. 

IPF Parameters 
(^cq> Gq) 

(%emirP Xemax) Image/PSNR 

BPP N, G, JPEG IPF EZW SPIHT 


1.00 (0.0469, 3.5938 barbl.O.jpg barbl.O.ipf N/A barbl.O.sp 

(-50, 451) 33.10 35.87 35.14 36.22 

9, 5 


TABLE 2(d) 15 

TABLE 3(d) 

PSNR improvement over baseline JPEG for 

[he 512 x 512 Barbara image. I PF PSNR dB improvement for 512 x 512 Barbara image . 


BPP 

^CBS AjPEG-ESAP 

^AQ 

^AQ-ESAP 

Ajo 

Ajo-esap 

^EZW 

BPP 

^IPF 

^EZW 

Aspiht 

0.25 

-0.01 0.71 

1.00 

1.94 

1.64 

2.03 

20 

1.75 

0.25 

2.21 

1.75 

2.38 

0.50 

-0.02 1.17 

1.72 

2.76 

2.36 

2.96 

2.26 

0.50 

2.82 

2.26 

2.98 

1.00 

-0.01 0.91 

2.12 

2.74 

2.84 

3.23 

2.04 

1.00 

2.77 

2.04 

3.12 


TABLE 3(a) 

JPEG, IPF, EZW and SPIHT comparison 512 x 512 Lena image. 
IPF Parameters 


(^cqj ®q) 

(Zemin. Zemax) toage/PSNR 


BPP 

N, G r 

JPEG 

IPF 

EZW 

SPIHT 

0.25 

(0.0469, 8.9609) 
(-94, 1106) 

13, 9.25 

lena.25.jpg 

31.68 

lena.25.ipf 

33.15 

N/A 

33.17 

lena.25.sp 

34.14 

0.50 

(0.0312, 7.8047) 
(-14, 976) 

13, 9.25 

lena.5.jpg 

34.90 

lena.5.ipf 

36.34 

N/A 

36.28 

lena.5.sp 

37.25 

1.00 

(0.0469, 8.4375) 
(-20, 1019) 

11, 10 

lenal.O.jpg 

37.96 

lenal.O.ipf 

39.28 

N/A 

39.55 

lenal.O.sp 

40.46 


TABLE 3(b) 


IPF PSNR dB improvement for 512 x 512 Lena image. 


BPP 

Aipp 

A EZW 

Aspiht 

0.25 

1.47 

1.49 

2.46 

0.50 

1.44 

1.38 

2.35 

1.00 

1.32 

1.59 

2.50 


To conclude this example, preprocessing the image to 
obtain a joint-optimized Q-table followed by ESAP post- 
processing generates an improvement A ao .esap °f 1-34 dB. 
For comparison, the last column shows the EZW A EZW 
improvement. For this case it is 1.49 dB. 

IPF Results 

This section compares IPF with baseline JPEG, EZW, and 
SPIHT algorithms. Tables 3(a) and ( b ) show the IPF results 
for the 512x512 8-BPP Lena image and Tables 3(c) and (d) 
35 show the results for the 512x512 Barbara image. All the 
images are .gif formatted. The IPF images were obtained 
with the algorithm described in FIG. 6. 

For comparison, the PSNR values obtained with baseline 
JPEG, Shapiro’s embedded zerotree wavelet coder, and 
40 experimental results obtained with Said and Pearlman’s set 
partitioning in hierarchical trees (SPIHT) subband coder are 
shown in the third and the two rightmost columns, respec- 
tively. The EZW PSNR values are taken from the literature 
and their corresponding images are not available. The values 
45 (oo c<? , G ) are the quantized cutoff frequencies and gains for 
the pre-post filters, respectively. The expanded dynamic 
range is bounded by (x emin ,x emax ) which can normally be 
represented by two 12-bit signed integers. “N” is the 2-D 
pre-post filters’ order NxN, and finally, “G” is the pre-post 
50 filters’ dB gain range (-G r . . . 0 . . . G r ). 


TABLE 3(c) 


JPEG, IPF, EZW and SPIHT comparison 
for 512 x 512 Barbara image. 


IPF Parameters 

(^cq? ®q) 

(Zemin. Zemax) Image/PSNR 


BPP N, G, JPEG 


0.25 (0.0469, 7.5938) barb.25.jpg 
(-195, 1021) 25.02 

9, 9 

0.50 (0.0312, 4.3125) barb.S.jpg 
(-33, 562) 28.27 

7, 6 


IPF 


barb.25.ipf 

27.23 

barb.5.ipf 

31.09 


EZW SPIHT 


N/A barb. 25. sp 
26.77 27.40 

N/A barb.5.sp 
30.53 31.25 


Alternate Embodiments 

FIG. 6 shows the IPF JPEG blocking noise reduction 
55 algorithm based on the dbx noise reduction system. In this 
case, the MMSE search looks for four parameters: the 
minimum and maximum pixel amplitudes (x e min , x e max ) and 
the (w,G) cutoff and gain parameters, respectively. 

In its simplest (sub-optimal) form, without MMSE 
go searching, these algorithms do not require any additional 
overhead. These algorithms may be implemented in firm- 
ware or a fast processor capable of real-time video sequence 
enhancement may be used to process these algorithms to 
accommodate improved performance over low bit rate con- 
55 nections. 

Variants of the disclosed ESAP system may be extended 
to include non-DCT coders; Including for example: 
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Vector Quantization (V Q); 

Projection Onto Convex Sets (POCS), and; 

Set Partitioning In Hierarchical Trees (SPIHT) octave-band 
subband coders. 

The ESAP and IPF algorithms of the present invention 5 
may be implemented in firmware to obtain fast real-time 
response; Either one may significantly enhance the visual 
quality of low bit rate Internet video or MPEG video 
sequences. 

This software has Potential applicability for video to 
enhancement of very low rate MPEG video sequences in the 
range of 28.8 kbps-56 kbps generally used in popular web 
browser streaming video viewers such as RealPlayer or 
Streamworks© and for H.261 ISDN coders from Picture- 
Tel® and Compression Labs. 15 

Other possible applications include use for moderate rate 
(4-8 Mbps) HDTV broadcasts. 

What is claimed is: 

1. A method for reducing image blocking artifacts result- 
ing from high JPEG image compression comprising the 20 
steps of: 

pre-emphasizing an original image file x with a filter P 
(w,G), to form a pre-emphasis signal X e ; 
wherein co is the HPF cutoff frequency corresponding 
to the separable DCT bandwidth / and where (to=/) 25 
and G is the pre-emphasis gain; 
compressing the dynamic range of said pre-emphasis 
signal X e into an 8-BPP range of 0-255 using a 
dynamic range compression (DRC) function to form a 
range-compressed signal X r ; 
using said range-compressed signal X r to compute a 
MMSE optimally quantized Q-table using a Lagrange 
multiplier minimization function adaptQw( ); 
using parameters compiled in said Q-table; 35 

compressing said range-compressed signal X r image file 
x using the JPEG standard IJG cjpeg( ) function to form 
a compressed JPEG file X,(k); 
de-compressing said JPEG file X,(k) to form a 
de-compressed image file x' ^ 40 

expanding said de-compressed image file using dynamic 
range expansion (DRE) to form an image file x e \ 
de-emphasizing said image file x e using a filter D(w, G) 
to create a x~ image; 45 

post-filtering said x~ image into an ximage, and thereafter; 
using an ESAP process to obtain further block smoothing 
of said x-image, wherein ESAP is Estimated Spectrum 
Adaptive Post-Filter. 

2. The method of claim 1, wherein the ESAP process 50 
comprises the steps of: 

providing a decompressed image file; 

providing dequantized DCT coefficients; 

estimating 2-D pixel-adaptive bandwidths directly from . . 

said dequantized DCT coefficients; and 
using said dequantized DCT coefficients to control a 2-D 
spatially-adaptive non-linear image post-filter. 
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3. The method of claim 2, further comprising the step of: 
using an edge detector to control a 2-D spatially-adaptive 

non-linear image post-filter. 

4 . The method of claim 2 , wherein the edge detector is 
Canny edge detector. 

5 . A method for reducing image blocking artifacts result- 
ing from high JPEG image compression comprising the 
steps of: 

pre-emphasizing an original image file x with a filter 
P(to,G), to form a pre-emphasis signal X e ; 
wherein co is the HPF cutoff frequency corresponding 
to the separable DCT bandwidth f and where (co^f) 
and G is the pre-emphasis gain; 
compressing the dynamic range of said pre-emphasis 
signal X e into an 8-EPP range of 0-255 using a 
dynamic range compression (DRC) function to form a 
range -compressed signal X r ; 
using said range-compressed signal X r to compute a 
MMSE optimally quantized Q-table using a Lagrange 
multiplier minimization function adaptQw( ); 
using parameters compiled in said Q-table; 
compressing said range-compressed signal X r image file 
x using the JPEG standard IJG cjpeg( ) function to form 
a compressed JPEG file X/k); 
de-compressing said JPEG file X,(k) to form a 
de-compressed image tile x" r ; 
expanding said de-compressed image file using dynamic 
range expansion (DRE) to form an image file x~ e ; 
de-emphasizing said image file x~ e using a filter D(co, G) 
to create a x" image; and 

using an Estimated Spectrum Adaptive Postfilter algo- 
rithm to post-filter said x image into an ximage. 

6. The method of claim 5, wherein the ESAP algorithm 
comprises the steps of: 

providing a decompressed image file; 
providing dequantized DCT coefficients; 
estimating 2-D pixel-adaptive bandwidths directly from 
said dequantized DCT coefficients; and 
using said dequantized DCT coefficients to control a 2-D 
spatially-adaptive non-linear image post-filter. 

7. The method of claim 6, further comprising the step of: 
using an edge detector to control a 2-D spatially-adaptive 

non-linear image post-filter. 

8. The method of claim 7, wherein the edge detector is 
Canny edge detector. 

9 . The method of claim 5 , further comprising the step of: 
using an Estimated Spectrum Adaptive Postfilter algo- 
rithm to post-filter said x",. image into an x r image. 

10 . The method of claim 5 , further comprising the step of: 
using an Estimated Spectrum Adaptive Postfilter algo- 
rithm to post-filter said x e image into an x e image. 





